Skip to content

Partitioning Rewrite - #394

Open
rsiminel wants to merge 41 commits into
devfrom
pynem-integration
Open

rsiminel wants to merge 41 commits into
devfrom
pynem-integration

Conversation

@rsiminel

@rsiminel rsiminel commented Aug 25, 2026 •

Copy link
Copy Markdown
Collaborator

Replaces PPanGGOLiN's vendored C NEM with a pynem dependancy. Implemented with help from Claude Code.

Why

The C NEM had to be compiled for every platform, read its input from files written to disk for every chunk, and computed in float32. On large pangenomes it returned a log-likelihood of −inf. This PR replaces it with pynem, installed from either PyPI or bioconda: a NumPy/numba implementation of the NEM algorithm. The C sources are deleted (−10,392 lines) and PPanGGOLiN now builds as one pure-Python wheel in a single CI job.

How

  • NEM input: the presence and coverage matrices are computed once per run, and each chunk's input is a column slice of them, passed to pynem in memory. No NEM files are written any more. --tmpdir and --keep_tmp_files are removed from ppanggolin partition, and --tmpdir from ppanggolin rarefaction.
  • Threads: each pool worker runs pynem's numba kernels and BLAS on one thread, so workers in parallel don't oversubscribe the cores. Numba uses its fork-safe threading layer, so pools forked after a NEM run don't hang.
  • Code structure: the NEM module is split into nem_input.py (building the input), model.py (the pynem wrapper), partition.py and rarefaction.py. Dataclasses replace long argument lists, and defaults are given only in argparse. No module-level state is shared between workers.
  • Dependancies: pynem[fast]>=0.6.1 (numba is now required) and threadpoolctl>=3.

Bugs

  • Rarefaction ignored --seed when K was given, forked workers shared one random state, and samples larger than the chunk size were drawn only from their last chunk. Each sample now has its own generator seeded from --seed.
  • Rarefaction samples depended on the order genomes were loaded; genomes are now sorted before drawing.
  • Every curve of the ICL figure was drawn twice.
  • Rarefaction overwrote the K range its caller passed.

Performance stats (measured on the PanGBank's refseq pangenomes)

  • Same partitions: up to 500 genomes, 0.07% of families end up in a different partition from PanGBank's, against 0.06% with the C NEM. Over 500 genomes, both differ on 0.9%. None switch between persistent and cloud.
  • Time: choosing K, pynem is 3.6× faster than the C NEM on one core, and 4.5× faster on eight over 500 genomes (medians).
  • Memory: up to 500 genomes on one core, the PSS peak is now 17% higher. Over 500 genomes on eight cores, it is 25% lower (medians over the whole ppanggolin partition command).

@JeanMainguy
JeanMainguy changed the base branch from master to dev August 26, 2026 08:30
@JeanMainguy
JeanMainguy marked this pull request as draft August 27, 2026 07:37
@github-actions github-actions Bot mentioned this pull request Sep 28, 2026
@rsiminel
rsiminel marked this pull request as ready for review November 5, 2026 16:41
@JeanMainguy
JeanMainguy requested a balanced review from Copilot November 10, 2026 16:04

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Thread limits are set only in worker initializers, leaving parent-process K scoring and whole-pangenome fitting unconstrained so even --cpu 1 can use all available cores.

1 open issue
What changed in this PR

Replaces PPanGGOLiN’s vendored C NEM implementation with in-memory pynem calls, simplifying packaging and reorganizing partitioning and rarefaction.

Changes:

  • Introduces reusable matrix inputs, a pynem wrapper, and dataclass-based options.
  • Revises rarefaction sampling, chunk processing, and plotting.
  • Removes native-extension build machinery and switches publishing to one pure-Python wheel.
File Description
tests/​conftest.py Removes obsolete partition global resets.
setup.py Deletes native-extension build configuration.
pyproject.toml Adds pynem and threadpoolctl; removes Cython.
ppanggolin/​workflow/​all.py Adopts partition and rarefaction option objects.
ppanggolin/​nem/​rarefaction.py Reworks sampling, in-memory partitioning, and plotting.
ppanggolin/​nem/​partition.py Reworks K estimation and chunk orchestration.
ppanggolin/​nem/​NEM/​nem_ver.h Removes legacy version declarations.
ppanggolin/​nem/​NEM/​nem_ver.c Removes legacy version reporting.
ppanggolin/​nem/​NEM/​nem_typ.h Removes legacy C types and constants.
ppanggolin/​nem/​NEM/​nem_stats.pyx Removes the Cython binding.
ppanggolin/​nem/​NEM/​nem_rnd.h Removes legacy random-function declarations.
ppanggolin/​nem/​NEM/​nem_rnd.c Removes legacy random routines.
ppanggolin/​nem/​NEM/​nem_nei.h Removes legacy neighborhood declarations.
ppanggolin/​nem/​NEM/​nem_nei.c Removes legacy neighborhood routines.
ppanggolin/​nem/​NEM/​nem_mod.h Removes legacy model declarations.
ppanggolin/​nem/​NEM/​nem_hlp.h Removes legacy help declarations.
ppanggolin/​nem/​NEM/​nem_hlp.c Removes legacy help output.
ppanggolin/​nem/​NEM/​nem_exe.h Removes the legacy C entry-point declaration.
ppanggolin/​nem/​NEM/​nem_alg.h Removes legacy algorithm declarations.
ppanggolin/​nem/​NEM/​lib_io.h Removes legacy I/O declarations.
ppanggolin/​nem/​NEM/​lib_io.c Removes legacy I/O routines.
ppanggolin/​nem/​NEM/​genmemo.h Removes legacy allocation declarations.
ppanggolin/​nem/​NEM/​genmemo.c Removes legacy allocation routines.
ppanggolin/​nem/​nem_input.py Adds matrix indexing, slicing, and smoothing graphs.
ppanggolin/​nem/​model.py Adds pynem fitting, labeling, and thread controls.
ppanggolin_env.yaml Adds threadpoolctl to the environment.
docs/​api/​ppanggolin.nem.md Documents the new model and input modules.
.github/​workflows/​pypi.yml Consolidates wheel and source-distribution builds.
.github/​workflows/​main.yml Removes deleted setup.py from workflow triggers.

Review effort: Balanced

Comment on lines +251 to +253
nem_input = build_nem_input(index, max_degree_smoothing)
if cpu <= 1:
return [score(nem_input, 0, free_dispersion, k) for k in ks]

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Resolved by limit main threads.

Comment thread ppanggolin/nem/partition.py Outdated
Comment on lines +500 to +510
with get_context("fork").Pool(
cpu, initializer=keep_index, initargs=(index,)
) as pool:
with tqdm(
total=len(chunks),
unit=" samples partitioned",
disable=disable_bar,
) as bar:
for result in pool.imap_unordered(_partition_chunk_worker, args):
votes.add(result.partitions)
bar.update()

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you can combine the two with statements

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Resolved by merge withs.

Comment thread ppanggolin/nem/partition.py Outdated
Comment on lines +258 to +263
with get_context("fork").Pool(
cpu, initializer=_keep_input, initargs=(nem_input,)
) as pool:
with tqdm(
total=len(ks), initial=1, unit="Number of partitions", disable=disable_bar
) as bar:

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same here. combine with statement so it is easier to read

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Resolved by merge withs.

Comment thread ppanggolin/nem/model.py Outdated
Comment on lines +16 to +18
# pynem 0.6.1's parallel kernels run on OpenMP, which kills workers forked after them
if "NUMBA_THREADING_LAYER" not in os.environ:
numba.config.THREADING_LAYER = "forksafe"

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This change globally the numba settings. not sure if it's a problem ? if ppanggolin is used as a library, importing it will change user numba setting which may be a problem?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Resolved by forksafe on use.

Comment thread ppanggolin/nem/model.py
from dataclasses import dataclass

# installed libraries
import numba

@JeanMainguy JeanMainguy Nov 16, 2026 •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

as numba is directly imported here it should be listed in the pyproject.toml explicitly

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Resolved by dependencies.

Comment thread ppanggolin_env.yaml
Comment on lines 17 to 23
- bokeh>=3.0.0,<3.4.0
- graph-tool>=2
- gb-io>=0.4.0
- threadpoolctl>=3
# Tools that are not in Python
- infernal=1
- mmseqs2=15

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pynem is not listed here but it should and maybe also add numba as it is imported by ppanggolin directly.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Resolved by dependencies.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants